Mapping components of a non-distributed environment to a distributed environment

ABSTRACT

Embodiments of the present invention disclose a method, a computer program product, and a computer system for mapping components of non-distributed environments to distributed environments. A computer receives a data pipeline configured for a non-distributed environment and identifies one or more bottleneck components of the data pipeline. In addition, the computer converts data used in the pipeline to a format compatible with a distributed environment and installs the necessary computing libraries necessary for operating the pipeline within the distributed environment. The computer further converts the code of the pipeline to a code that is compatible with the distributed environment and optimizes components of the pipeline for use in the distributed environment.

BACKGROUND

The present invention relates generally to data analytics, and moreparticularly to mapping components of a non-distributed environment to adistributed environment.

In general, scaling a technological solution beyond a certain pointrequires the adoption of an environment with multiple machines because asingle machine does not have the computational resources to processincreasing amounts of data. Most existing solutions typically start offdirectly with utilization of a pure non-distributed environment or apure distributed environment, but few tackle the process of creating aframework for converting legacy components of a non-distributedenvironment to components of a distributed environment.

SUMMARY

Embodiments of the present invention disclose a method, a computerprogram product, and a computer system for mapping components of datapipeline from a non-distributed network to a distributed network.

According to an embodiment, the method comprises a computer identifyingat least one bottleneck component of one or more components comprisingthe data pipeline when implemented within the non-distributed computingenvironment, and the computer converting data corresponding to the atleast one bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment. In addition, the method comprises thecomputer installing libraries corresponding to an architecture of thedistributed computing environment, as well as the computer convertingcode of the at least one bottleneck component from a format compatiblewith the non-distributed computing environment to a format compatiblewith the distributed computing environment.

In embodiments, the method further comprises the computer measuring aperformance of the at least one bottleneck component when implementedwithin the distributed computing environment.

Moreover, in some embodiments, the method of identifying the at leastone bottleneck component further comprises at least one of the computeridentifying at least one component of the one or more components havinga greatest memory usage, and the computer identifying at least onecomponent of the one or more components having a memory usage exceedinga threshold value.

In further embodiments, the method of identifying the at least onebottleneck component further comprises at least one of the computeridentifying at least one component of the one or more components takinga longest time to complete a task, and the computer identifying at leastone component of the one or more components taking longer than athreshold time to complete a task.

In yet further embodiments, the method may include the computergenerating a model for operating the data pipeline having one or moreparameters and one or more constraints, wherein the one or moreparameters include memory usage and time. The method may furthercomprise the computer determining a lowest memory usage and shortesttime based on the model, and the computer identifying at least onecomponent of the one or more components that fails to perform at thelowest memory usage or the shortest time.

In still yet further embodiments, determining the at least onebottleneck component in future iterations is based on the measuredperformance of the at least one bottleneck component.

In some embodiments, converting code of the at least one bottleneckfurther comprises the computer determining that code of the at least onebottleneck component is incompatible with code of the non-distributedcomputing environment, and the computer generating a wrapper functionenabling compatibility between code of the at least one bottleneckcomponent and code of the non-distributed computing environment.

According to embodiments, a computer program product is provided. Thecomputer program product comprises program instructions to identify atleast one bottleneck component of one or more components comprising thedata pipeline when implemented within the non-distributed computingenvironment, and program instructions to convert data corresponding tothe at least one bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment. In addition, the computer programproduct may also comprise program instructions to install librariescorresponding to an architecture of the distributed computingenvironment, and program instructions to convert code of the at leastone bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment.

In embodiments, the computer program product may further compriseprogram instructions to measure a performance of the at least onebottleneck component when implemented within the distributed computingenvironment.

Moreover, in some embodiments, program instructions to identify the atleast one bottleneck component further comprise at least one of programinstructions to identify at least one component of the one or morecomponents having a greatest memory usage, and program instructions toidentify at least one component of the one or more components having amemory usage exceeding a threshold value.

In further embodiments, program instructions to identify the at leastone bottleneck component further comprise at least one of programinstructions to identify at least one component of the one or morecomponents taking a longest time to complete a task, and programinstructions to identify at least one component of the one or morecomponents taking longer than a threshold time to complete a task.

In yet further embodiments, program instructions to identify the atleast one bottleneck component further comprise program instructions togenerate a model for operating the data pipeline having one or moreparameters and one or more constraints, wherein the one or moreparameters include memory usage and time. In addition, the computerprogram product may further include program instructions to determine alowest memory usage and shortest time based on the model, and programinstructions to identify at least one component of the one or morecomponents that fails to perform at the lowest memory usage or theshortest time.

In still yet further embodiments, program instructions to determine theat least one bottleneck component in future iterations is based on themeasured performance of the at least one bottleneck component.

In some embodiments, program instructions to convert code of the atleast one bottleneck further comprise program instructions to determinethat code of the at least one bottleneck component is incompatible withcode of the non-distributed computing environment, and programinstructions to generate a wrapper function enabling compatibilitybetween code of the at least one bottleneck component and code of thenon-distributed computing environment.

According to some embodiments, a computer system is provided. Thecomputer system comprises program instructions to identify at least onebottleneck component of one or more components comprising the datapipeline when implemented within the non-distributed computingenvironment, and program instructions to convert data corresponding tothe at least one bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment. In addition, the computer system mayfurther comprise program instructions to install libraries correspondingto an architecture of the distributed computing environment, and programinstructions to convert code of the at least one bottleneck componentfrom a format compatible with the non-distributed computing environmentto a format compatible with the distributed computing environment.

In embodiments, the computer system may further comprise programinstructions to measure a performance of the at least one bottleneckcomponent when implemented within the distributed computing environment.

Moreover, in some embodiments, program instructions to identify the atleast one bottleneck component further comprise at least one of programinstructions to identify at least one component of the one or morecomponents having a greatest memory usage, and program instructions toidentify at least one component of the one or more components having amemory usage exceeding a threshold value.

In further embodiments, program instructions to identify the at leastone bottleneck component further comprise at least one of programinstructions to identify at least one component of the one or morecomponents taking a longest time to complete a task, and programinstructions to identify at least one component of the one or morecomponents taking longer than a threshold time to complete a task.

In yet further embodiments, program instructions to identify the atleast one bottleneck component further comprise program instructions togenerate a model for operating the data pipeline having one or moreparameters and one or more constraints, wherein the one or moreparameters include memory usage and time. In addition, the computersystem may further provide program instructions to determine a lowestmemory usage and shortest time based on the model, and programinstructions to identify at least one component of the one or morecomponents that fails to perform at the lowest memory usage or theshortest time.

In still yet further embodiments, program instructions to determine theat least one bottleneck component in future iterations is based on themeasured performance of the at least one bottleneck component.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The following detailed description, given by way of example and notintended to limit the invention solely thereto, will best be appreciatedin conjunction with the accompanying drawings, in which:

FIG. 1 depicts a schematic diagram of an environment mapping system 100,in accordance with an embodiment of the present invention.

FIG. 2 depicts a flowchart illustrating the operations of theenvironment mapper 142 of environment mapping system 100 in mapping datapipeline components from a non-distributed environment to a distributedenvironment, in accordance with an embodiment of the present invention.

FIG. 3 depicts an example data pipeline, in accordance with anembodiment of the present invention.

FIG. 4 depicts a block diagram depicting the hardware components ofenvironment mapping system 100 of FIG. 1, in accordance with anembodiment of the present invention.

FIG. 5 depicts a cloud computing environment, in accordance with anembodiment of the present invention.

FIG. 6 depicts abstraction model layers, in accordance with anembodiment of the present invention.

The drawings are not necessarily to scale. The drawings are merelyschematic representations, not intended to portray specific parametersof the invention. The drawings are intended to depict only typicalembodiments of the invention. In the drawings, like numbering representslike elements.

DETAILED DESCRIPTION

Detailed embodiments of the claimed methods, computer program products,and computer systems are disclosed herein; however, it can be understoodthat the disclosed embodiments are merely illustrative of the claimedmethods, computer program products, and computer systems that may beembodied in various forms. This invention may, however, be embodied inmany different forms and should not be construed as limited to theexemplary embodiments set forth herein. Rather, these exemplaryembodiments are provided so that this disclosure will be thorough andcomplete and will fully convey the scope of this invention to thoseskilled in the art. In the description, details of well-known featuresand techniques may be omitted to avoid unnecessarily obscuring thepresented embodiments.

References in the specification to “one embodiment”, “an embodiment”,“an example embodiment”, etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Furthermore, when a particularfeature, structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to affect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

In the interest of not obscuring the presentation of embodiments of thepresent invention, in the following detailed description, someprocessing steps or operations that are known in the art may have beencombined together for presentation and for illustration purposes and insome instances may have not been described in detail. In otherinstances, some processing steps or operations that are known in the artmay not be described at all. It should be understood that the followingdescription is rather focused on the distinctive features or elements ofvarious embodiments of the present invention.

As data collection increases, the performance of particular componentsin a non-distributed pipeline may decrease due to, for example, aninability of the non-distributed environment to handle large amounts ofdata or complex computations thereof. In many cases, it is more costeffective to purchase additional, average-powered computing devicesrather than purchase a single, high-powered computing device. For thatreason, it may be more efficient, in terms of both cost and computingpower, to translate components of a pipeline that were originallyconfigured for a non-distributed environment into components configuredfor use in a distributed environment. Moreover, in many cases,distributing a pipeline across the computing power of many machines canoften times result in an increase in pipeline performance, despite theaggregate power of the distributed machines merely equaling that of thesingle machine.

Environment mapping system 100, in accordance with an embodiment of theinvention, is illustrated by FIG. 1. In the example embodiment,environment mapping system 100 includes computing device 110,non-distributed environment 120, distributed environment 130, and server140, all interconnected via network 108.

In the example embodiment, network 108 is a communication channelcapable of transferring data between connected devices. In the exampleembodiment, network 108 may be the Internet, representing a worldwidecollection of networks and gateways to support communications betweendevices connected to the Internet. Moreover, network 108 may include,for example, wired, wireless, or fiber optic connections which may beimplemented as an intranet network, a local area network (LAN), a widearea network (WAN), or any combination thereof. In further embodiments,network 108 may be a Bluetooth network, a WiFi network, or a combinationthereof. In yet further embodiments, network 108 may be atelecommunications network used to facilitate telephone calls betweentwo or more parties comprising a landline network, a wireless network, aclosed network, a satellite network, or any combination thereof. Ingeneral, network 108 can be any combination of connections and protocolsthat will support communications between computing device 110,non-distributed environment 120, distributed environment 130, and server140.

In the example embodiment, computing device 110 includes user interface112. Computing device 110 may be a server, laptop computer, a notebook,a tablet computer, a netbook computer, a personal computer (PC), adesktop computer, a personal digital assistant (PDA), a rotary phone, atouchtone phone, a smart phone, a mobile phone, a virtual device, a thinclient, or any other electronic device or computing system capable ofreceiving and sending data to and from other computing devices. Whilecomputing device 110 is shown as a single device, in other embodiments,computing device 110 may be comprised of a cluster or plurality ofcomputing devices, working together or working separately. Computingdevice 110 is described in more detail with reference to FIG. 4.

User interface 112 is a software application which allows a user ofcomputing device 110 to interact with computing device 110 as well asother connected devices via network 108. User interface 112 may beconnectively coupled to hardware components, such as those depicted byFIG. 4, for receiving user input, including mice, keyboards,touchscreens, microphones, cameras, and the like. In embodiments, userinterface 112 may be implemented via a web browsing applicationcontaining a graphical user interface (GUI) that is capable oftransferring data files, folders, audio, video, hyperlinks, compresseddata, and other forms of data transfer individually or in bulk.Alternatively, user interface 112 may be implemented via otherintegrated or standalone software applications and hardware capable ofreceiving user interaction and communicating with other electronicdevices.

Non-distributed environment 120 includes server 122. In the exampleembodiment, non-distributed environment 120 is an environment in whichall data storage and processing are all performed on individualcomputing devices that do not collaboratively work together to performone or more tasks.

Server 122 may be a server, laptop computer, a notebook, a tabletcomputer, a netbook computer, a personal computer (PC), a desktopcomputer, a personal digital assistant (PDA), a rotary phone, atouchtone phone, a smart phone, a mobile phone, a virtual device, a thinclient, or any other electronic device or computing system capable ofreceiving and sending data to and from other computing devices. Whileserver 122 is shown as a single device, in other embodiments, server 122may be comprised of a cluster or plurality of computing devices, workingtogether or working separately. Server 122 is described in more detailwith reference to FIG. 4.

Distributed environment 130 includes server 132, server 134, and server136. In the example embodiment, distributed environment 130 is anenvironment in which data storage and processing is performed across twoor more computing devices working collaboratively within a network toperform one or more tasks.

In the example embodiment, server 132, server 134, and server 136 mayeach be a server, laptop computer, a notebook, a tablet computer, anetbook computer, a personal computer (PC), a desktop computer, apersonal digital assistant (PDA), a rotary phone, a touchtone phone, asmart phone, a mobile phone, a virtual device, a thin client, or anyother electronic device or computing system capable of receiving andsending data to and from other computing devices. In the exampleembodiment, server 132, server 134, and server 136 are configured tocommunicate with one another such that data, programming, and processingmay be shared amongst one another. While in the example embodimentdistributed environment 130 includes server 132, server 134, and server136, in other embodiments, distributed environment 130 may contain anynumber of two or more computing devices. Server 132, server 134, andserver 136 are described in more detail with reference to FIG. 4.

In the example embodiment, non-distributed environment 120 has acomputing power that is equal to that of distributed environment 130,i.e., server 122 has a computing power equal to that of the aggregate ofserver 132, server 134, and server 136. For example, server 122 may be amachine with 48 cores and 180 GB of storage while server 132, server134, and server 136 may each have 16 cores and 60 GB of storage,respectively. In other embodiments, however, non-distributed environment120 may have more or less computing power than that of distributedenvironment 130.

In the example embodiment, server 140 includes environment mapper 142and may be a server, laptop computer, a notebook, a tablet computer, anetbook computer, a personal computer (PC), a desktop computer, apersonal digital assistant (PDA), a rotary phone, a touchtone phone, asmart phone, a mobile phone, a virtual device, a thin client, or anyother electronic device or computing system capable of receiving andsending data to and from other computing devices. While server 140 isshown as a single device, in other embodiments, server 140 may becomprised of a cluster or plurality of computing devices, workingtogether or working separately. Server 140 is described in more detailwith reference to FIG. 4.

Environment mapper 142 is a software application capable of receiving adata pipeline configured for non-distributed environment 120 andidentifying one or more bottleneck components of the data pipeline. Inaddition, environment mapper 142 is capable of converting data used inthe non-distributed environment to a format compatible with distributedenvironment 130 and installing the computing libraries necessary foroperating the pipeline within distributed environment 130. Environmentmapper 142 is further capable of converting programming language, suchas object code or source code, of the pipeline to programming languagethat is compatible with distributed environment 130 and optimizingcomponents of the data pipeline for use in distributed environment 130.

FIG. 2 illustrates the operations of environment mapper 142 ofenvironment mapping system 100 in mapping components of a pipelinewithin a non-distributed environment to a distributed environment.

Environment mapper 142 receives a data pipeline configured fornon-distributed environment 120 (step 202). In the example embodiment,environment mapper 142 is linked to non-distributed environment 120 inorder to identify manners by which one or more components of the datapipeline may be mapped for use in distributed environment 130. Inembodiments, for example, environment mapper 142 may be linked tonon-distributed environment 120 via network 108 to remotely monitorperformance of the components comprising the data pipeline.Alternatively, in another example, environment mapper may receiveperformance data, either past or in real time, for components of thedata pipeline via network 108 for remote analysis. In a yet furtherexample, environment mapper 142 may be incorporated into non-distributedenvironment 120 in order to locally analyse components of the datapipeline. According to the invention herein, such pipelines may be anysequence of functional components, modules, operations, etc., in whichone component leads to the next, with each component configured forperforming a particular operation. For example, components of a datapipeline may include data quality checks, data processing, modelgeneration, model training, and the like. Moreover, it will beappreciated that the present invention may be applicable to datapipelines having any number of components or modules that are configuredfor operation in non-distributed environment 120.

With reference to an illustrative example depicted by FIG. 3, forinstance, environment mapper 142 is linked via network 108 to a pipelineimplemented within non-distributed environment 120 configured forgenerating and evaluating models that describe healthcare claims data.As depicted by FIG. 3, the data pipeline comprises the components ingestdata, data quality checks, data preparation, create training dataset,create test dataset, train model, and evaluate model.

Referring now back to FIG. 2, environment mapper 142 identifies one ormore bottleneck components in the received data pipeline ofnon-distributed environment 120 (step 204). In the example embodiment,bottleneck components are the components slowing, reducing, or otherwiseimpeding the pipeline from performing more efficiently. As describedabove, components are functional modules and environment mapper 142identifies the one or more bottleneck components of the pipeline usingabsolute or relative thresholds of what is considered suboptimalperformance for each component of the pipeline. In order to establishthe thresholds that distinguish optimal performance from suboptimalperformance, environment mapper 142 may be configured to referenceprevious measurements, receive user input thresholds, or a combinationof analysis methods. For example, environment mapper 142 may beconfigured to periodically measure the performance of a particularcomponent at a prescribed frequency and identify suboptimal performancebased on threshold variations in the measurements as data is processed,for example based on a standard deviation. Alternatively, environmentmapper 142 may be configured to analyse performance of a particularcomponent across multiple datasets to determine whether the componentresponds to additional data, computations, increasing complexity, etc.,as expected. For example, environment mapper 142 may vary data, e.g.double an amount, and determine whether a proportional variation isobserved within the performance, e.g. double time to compute. If theresults indicate a lesser performance than expected, e.g. three times aslong to compute, then environment mapper 142 identifies the component asa bottleneck. In yet further embodiments, environment mapper 142 may beconfigured to identify suboptimal performance of a component byanalysing data of the component within a distributed environment andcomparing the observed performance to that of a similar component in anon-distributed environment. For example, the performance of aparticular pipeline component implemented in non-distributed environment120 may be compared to that of a similar pipeline component beingimplemented in distributed environment 130.

The aforementioned methods, as well as many others, can be performedusing several approaches, including a memory driven approach, aperformance driven approach, and a hybrid approach, all described below(step 204 continued). In the following approaches, it should be notedthat absolute and relative thresholds can be specific to each individualcomponent, groups of components, or generic to the entire data pipeline.In addition, in some embodiments, duration thresholds may be furtherassociated with each threshold such that brief spikes in measurements donot result in false positives. For example, environment mapper 142 mayrequire a measurement exceed a threshold for one minute or longer inorder to trigger identification of a bottleneck component.

In order to identify bottleneck components using a memory drivenapproach (step 204 continued), environment mapper 142 uses tools such astop command and other memory commands to identify which components aretaking a longest time to perform a task. Top command displays processorand memory activity for running processes that can be analysed todetermine which components are performing sub-optimally based onabsolute or relative thresholds. In the example above, for instance,environment mapper 142 may set an absolute processor use threshold of70% for any component of the pipeline, indicating that componentsutilizing 70% or more of the processor when implemented withinnon-distributed environment 120 are to be considered a bottleneck in thepipeline. Similarly, environment mapper 142 may receive an absoluteprocessor use threshold of 50K (50 kilobytes) when implemented withinnon-distributed environment 120, indicating that a component is abottleneck if processor use exceeds 50K. Alternatively, in anotherexample, environment mapper 142 may determine processor use for allcomponents of the pipeline and identify the component have a relativelyhighest processor use as a bottleneck, for example by comparing the oneor more components and designating the component taxing the processorthe most as the bottleneck. In a similar example, environment mapper 142may be further configured to only identify a component having a highestprocessor use as a bottleneck if the use exceeds a second highest oraverage processor use of the remaining components by a specificthreshold. To that point, environment mapper 142 may be configured toidentify components having a highest processing use as bottleneck onlyif the use exceeds the next highest processor using component, oraverage of all remaining components, by 20%. As previously mentioned,thresholds for each component may be configured for each particularcomponent as well as groups of components. In some embodiments, athreshold can be associated with the entire data pipeline, withperformance in excess of the threshold indicating that all components ofthe data pipeline are bottlenecks requiring remapping.

In order to identify bottleneck components using performance drivenidentification (step 204 continued), environment mapper 142 utilizessystem timing commands to identify statistics describing how longvarious functions take to perform. In some embodiments, environmentmapper 142 may utilize profiling tools to establish a profile of eachcomponent that provides a set of statistics describing how often and forhow long various components are taking at a per systemcall/line/function/module level. In the example above, for instance,environment mapper 142 may receive a one-hour time threshold applicableto every component in the pipeline, indicating that if any particularcomponent exceeds an hour in time to complete a task, the component isidentified as a bottleneck. Alternatively, and using a relativethreshold, environment mapper 142 may be configured to identify anycomponent which takes double an amount of time as the shortest durationcomponent in the pipeline as being bottleneck components. In addition,environment mapper 142 may be configured for determining componentsoperating more frequently than an absolute or relative threshold value,for example a component utilizing processing power more than five timesper hour or more than double a frequency of a next highest component. Ingeneral, environment mapper 142 may utilize and analyse various metrics,for example combining both frequency and duration of operationsperformed by a component, in order to identify bottlenecks.

In some embodiments (step 204 continued), environment mapper 142 mayidentify bottleneck components using a hybrid approach of both memoryand performance driven identification. In such embodiments, environmentmapper 142 may set thresholds through parameters for each component tocomplete a task. Such thresholds may include a maximum memory availablefor a component within the pipeline and/or a maximum acceptable time forcompletion of a component within the data pipeline. Moreover,environment mapper 142 may further automate experiments that identify anoptimal memory and time that can be met by the system with theconstraints provided. For example, if a specific memory usage isallotted to a particular component to ensure other resources in theenvironment have ample memory access, a maximum memory threshold may beprovided as a constraint on that component. Alternatively, if a fixedtime threshold is crucial, e.g. no more than 2 hours for a component tocomplete its function, then a time threshold for that component orpipeline may be provided. In embodiments where both memory and time areimportant, they can both be provided as parameters. If this threshold isfixed and well defined, the environment mapper can encode the logic andautomatically identify bottleneck components.

It is important to note that another consideration in determiningbottlenecks is an effectiveness of distributing the component withindistributed environment 130. For instance, components which can make useof parallel processing, such as learners utilizing iterative algorithmsthat require intermediate storage, are more suitable for translation toa distributed environment than components which are incapable or have noneed for parallelization, such as data preparation. Thus, althoughenvironment mapper 142 may identify some components as bottlenecks, thecomponents may not be the most suitable candidate for mapping to aconfiguration compatible with distributed environment 130. Accordingly,in some embodiments, a user may be prompted after identification of thebottleneck components to affirmatively select one or more components tomap. In such embodiments, the components may be ranked based on lowestrelative performance and listed in descending order for a user to selectvia user interface 112. In further embodiments, environment mapper 142may recall previously component mappings and results to indicate whichcomponents of those identified as bottleneck components map better todistributed environment 130 based on past experience, and continuouslyimprove component suggestions as more data is accumulated. In yetfurther embodiments, environment mapper 142 may autonomously determinewhich components to map to distributed environment 130 based on acombination of those components identified as bottlenecks above,translatability of the bottleneck components to distributed environment130, and analysis of previously mapped components. It should be notedthat the present invention is not limited to mapping only a singlebottleneck component to distributed environment 130, and in some casesmay even identify every component of the data pipeline as a bottleneck.

Continuing the example illustrated by FIG. 3 having components ingestdata, data quality checks, data preparation, create training dataset,create test dataset, train model, and evaluate model, environment mapper142 performs a performance based and memory driven analysis to determinethat the data preparation and train model components are bottlenecks.Moreover, environment mapper 142 additionally considers that the trainmodel component, unlike the data preparation component, utilizeslogistic regression which benefits from usage of parallel in-memory dataprocessing. By contrast, the data preparation component may havefunctions that are highly dependent on the results of prior qualitychecks, hence the availability of a higher degree of parallelizationwouldn't benefit the data preparation component much. Thus, consideringboth performance and memory based analysis, as well as translatabilityof the components, environment mapper 142 identifies the train modelcomponent as the most suitable bottleneck component for mapping todistributed environment 130. Note that in some embodiments, both thedata preparation and train model components may be mapped to distributedenvironment 130 through, for example, user preferences, manual override,etc.

Having identified bottleneck components of non-distributed environment120, environment mapper 142 loads data of non-distributed environment120 into distributed environment 130 (step 206). In embodiments, thedata used in the pipeline when implemented in non-distributedenvironment 120 must be modified for use in distributed environment 130because the data had previously only been configured for access by asingle device. In distributed environment 130, the data must be modifiedsuch that multiple computing devices may access and make use of it,often times simultaneously. Accordingly, when applicable, environmentmapper 142 is configured to replicate the data across multiple storagesites in order to make the data more accessible to additional computingdevices within distributed environment 130, such as server 132, server134, and server 136. Moreover, storing the data across multiplecomputing devices provides backup mechanisms in the case a storage nodeis destroyed, gets corrupted, loses power, etc. In addition, becausemoving from a non-distributed environment to a distributed environmentoften times requires programming and functionality not generallyincluded in a pipeline implemented within a non-distributed environment,loading the data into distributed environment 130 may further compriseenvironment mapper 142 converting the data to a format compatible withthe desired architecture of distributed environment 130.

In the example introduced above and depicted by FIG. 3, for instance,where train model has been identified as a bottleneck component,environment mapper 142 replicates the health claims data from server 122of non-distributed environment 120 across server 132, server 134, andserver 136 of distributed environment 130. In addition, environmentmapper 142 further converts the health claims data from a relationaldatabase to a file on disk.

Environment mapper 142 installs necessary computing libraries foroperation of the bottleneck component(s) within distributed environment130 (step 208). As mentioned above, different distributed environmentarchitectures may call for different file types, formats, and othercompatibility necessities in order to function properly. Similarly,different architectures may further require, or simply benefit from,integrated libraries which contain a collection of precompiled data,such as routines and reference resources, that a component or pipelinecan use. Libraries are advantageous for storing frequently used routinesand data because they do not need to be explicitly linked to everycomponent or pipeline that uses them. Rather, architectures haveintegrated libraries, from which the routines and data may be called bypipelines and components thereof utilizing the architecture. Suchlibraries may include data such as configuration data, documentation,etc., while routines may include pre-written code and subroutines forperforming various operations. In the example embodiment, environmentmapper 142 installs the necessary libraries for the particulararchitecture of distributed environment 130 on at least one computingdevice of distributed environment 130 (e.g. server 134). In otherembodiments, however, environment mapper 142 may install the librarieson multiple or all the computing devices of distributed environment 130(e.g. server 132, server 134, and server 136).

With reference again to the example above, environment mapper 142installs libraries corresponding to the desired architecture ofdistributed environment 130, which include configuration data andsubroutines for data quality checks.

Environment mapper 142 converts programming language corresponding toone or more components of the pipeline to programming language that issuitable for the architecture of distributed environment 130 (step 210).Like the data format and libraries described above which are specific toan architecture, as may be a programming language for each architecturetype. When the programming language of the bottleneck component(s), orof other components within the data pipeline, differs from that of thearchitecture for distributed environment 130, environment mapper 142 mayneed modify the programming language of the components. In someembodiments, only programming language of the bottleneck component beingmapped to distributed environment 130 need be modified. In embodiments,environment mapper 142 may be configured to convert the coding of thebottleneck component in entirety to conform with that of distributedenvironment 130. In other embodiments, environment mapper 142 may becapable of adapting the bottleneck component to distributed environment130 using other means, for example through use of a wrapper function. Awrapper function is a subroutine whose main purpose is to call a secondsubroutine. In the present invention, a wrapper function may be writtento adapt the native programming language of the bottleneck component tothat of distributed environment 130. Thus, rather than modifying theentirety native code used by the component itself, environment mapper142 simply generates a wrapper function suitable for translating the twoprogramming languages. While this method and others may be practical inadapting some programming languages to others, it may not be suitable inall situations and thus, in some embodiments, environment mapper 142converts the programming language of all components of the pipeline tothat of distributed environment 130. In some embodiments, environmentmapper 142 may be configured to first determine whether the programminglanguage of the one or more bottleneck components are compatible withthe desired architecture of distributed environment 130 and, if not,determine whether a wrapper function can reconcile the inconsistentprogramming languages. Then, only if environment mapper 142 is incapableof either/both modifying the programming language of only the bottleneckcomponent(s) or generating a file wrapper for the bottleneckcomponent(s) does environment mapper 142 convert the programminglanguage of components comprising the whole pipeline to that ofdistributed environment 130.

In the example introduced earlier, for example, in which the train modelcomponent was identified as a bottleneck, environment mapper 142converts the programming language for the component into that which issuitable for use with the desired architecture for distributedenvironment 130. Alternatively, environment mapper 142 generates awrapper function for the component train model that is suitable for thedesired architecture of distributed environment 130.

Environment mapper 142 measures and optimizes the components of thepipeline when implemented within distributed environment 130 (step 212).In the example embodiment, environment mapper 142 utilizes performancemeasuring tools to determine a performance of the mapped components andpipeline as a whole when implemented in distributed environment 130. Forexample, environment mapper 142 may compare performance of thedistributed pipeline and/or bottleneck component(s) to that measuredwhen previously determining which components of the non-distributedpipeline are bottlenecks. Alternatively, environment mapper 142 maycompare performance of the distributed pipeline and/or bottleneckcomponents to that of known benchmarks, for example comparablepipelines/components, or to anticipated results. In the exampleembodiment, such measurements are recorded and later used in evaluatingthe effectiveness of the mapping. For example, environment mapper 142may record which and how components of the pipeline are mapped todistributed environment 130, as well as a corresponding change inperformance of the component(s) and pipeline as a whole. These recordsmay then be utilized as a resource in future mappings, for example indetermining which bottleneck component to map, how many bottleneckcomponents to map, whether mapping is practical/feasible, whether to usea wrapper function, which programming language yields best results for aparticular architecture, which libraries to install, and otherconsiderations made by environment mapper 142. Such insights may beextracted from the recorded data using known data analysis techniquesand future insights may be extracted using cognitive learning techniquesand artificial intelligence. In addition, environment mapper 142 mayfurther optimize the pipeline and components thereof where possible byusing in-memory data process when applicable, for example in machinelearning algorithms. Such optimization techniques similarly be recordedand used in determining a most effective means for future componentmappings.

Consider the pipeline of the example pipeline introduced above, forinstance, where the train model component has been mapped to distributedenvironment 130. Environment mapper 142 measures the performance of thetrain model component and pipeline as a whole and compares it to themeasured performances prior to the mapping, as well as expectedperformance. Environment mapper 142 records the performance data, forexample a 20% increase in overall pipeline efficiency, and utilizes thedata in future mappings in determining which components are mostsuitable for translation. If, for example, environment mapper 142previously recorded that for a similar pipeline mapping evaluate modelto distributed environment 130 resulted in a 30% increase in overallpipeline efficiency, environment mapper 142 may alternatively map theevaluate model component, or both, in future mappings.

FIG. 3 depicts an example data pipeline comprising the components ingestdata, data quality checks, data preparation, create training dataset,create test dataset, train model, and evaluate model.

Based on the foregoing, a computer system, method, and computer programproduct have been disclosed. However, numerous modifications andsubstitutions can be made without deviating from the scope of thepresent invention. Therefore, the present invention has been disclosedby way of example and not limitation.

FIG. 4 depicts a block diagram of computing device 110 of environmentmapping system 100 of FIG. 1, in accordance with an embodiment of thepresent invention. It should be appreciated that FIG. 4 provides only anillustration of one implementation and does not imply any limitationswith regard to the environments in which different embodiments may beimplemented. Many modifications to the depicted environment may be made.

Computing device 110 may include one or more processors 02, one or morecomputer-readable RAMs 04, one or more computer-readable ROMs 06, one ormore computer readable storage media 08, device drivers 12, read/writedrive or interface 14, network adapter or interface 16, allinterconnected over a communications fabric 18. Communications fabric 18may be implemented with any architecture designed for passing dataand/or control information between processors (such as microprocessors,communications and network processors, etc.), system memory, peripheraldevices, and any other hardware components within a system.

One or more operating systems 10, and one or more application programs11, for example environment mapper 142, are stored on one or more of thecomputer readable storage media 08 for execution by one or more of theprocessors 02 via one or more of the respective RAMs 04 (which typicallyinclude cache memory). In the illustrated embodiment, each of thecomputer readable storage media 08 may be a magnetic disk storage deviceof an internal hard drive, CD-ROM, DVD, memory stick, magnetic tape,magnetic disk, optical disk, a semiconductor storage device such as RAM,ROM, EPROM, flash memory or any other computer-readable tangible storagedevice that can store a computer program and digital information.

Computing device 110 may also include a R/W drive or interface 14 toread from and write to one or more portable computer readable storagemedia 26. Application programs 11 on said devices may be stored on oneor more of the portable computer readable storage media 26, read via therespective R/W drive or interface 14 and loaded into the respectivecomputer readable storage media 08.

Computing device 110 may also include a network adapter or interface 16,such as a TCP/IP adapter card or wireless communication adapter (such asa 4G wireless communication adapter using OFDMA technology). Applicationprograms 11 on said computing devices may be downloaded to the computingdevice from an external computer or external storage device via anetwork (for example, the Internet, a local area network or other widearea network or wireless network) and network adapter or interface 16.From the network adapter or interface 16, the programs may be loadedonto computer readable storage media 08. The network may comprise copperwires, optical fibers, wireless transmission, routers, firewalls,switches, gateway computers and/or edge servers.

Computing device 110 may also include a display screen 20, a keyboard orkeypad 22, and a computer mouse or touchpad 24. Device drivers 12interface to display screen 20 for imaging, to keyboard or keypad 22, tocomputer mouse or touchpad 24, and/or to display screen 20 for pressuresensing of alphanumeric character entry and user selections. The devicedrivers 12, R/W drive or interface 14 and network adapter or interface16 may comprise hardware and software (stored on computer readablestorage media 08 and/or ROM 06).

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 5, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 40 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 40 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 5 are intended to be illustrative only and that computing nodes40 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 6, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 5) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 6 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and mapping processing 96.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The invention claimed is:
 1. A method for mapping one or more componentsof a data pipeline from a non-distributed computing environment to adistributed computing environment, the method comprising: a computeridentifying at least one bottleneck component of one or more componentscomprising the data pipeline when implemented within the non-distributedcomputing environment, wherein identifying the at least one bottleneckcomponent is based on a hybrid approach, wherein the hybrid approachsets thresholds through parameters for each component to complete atask, and further comprises: increasing an input to the one or morecomponents comprising the data pipeline; determining that a performanceof the at least one component is less than proportional to the increaseof the input; the computer converting data corresponding to the at leastone bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment; the computer installing librariescorresponding to an architecture of the distributed computingenvironment; and the computer converting code of the at least onebottleneck component from a format compatible with the non-distributedcomputing environment to a format compatible with the distributedcomputing environment.
 2. The method of claim 1, further comprising: thecomputer measuring a performance of the at least one bottleneckcomponent when implemented within the distributed computing environment;and wherein identifying the at least one bottleneck component of one ormore components comprising the data pipeline is further based on themeasured performance of the at least one bottleneck.
 3. The method ofclaim 2, further comprising: the computer determining whether the atleast one bottleneck component is capable of being parallelized; andwherein identifying the at least one bottleneck component of one or morecomponents comprising the data pipeline is further based on determiningthat the at least one bottleneck component is capable of beingparallelized.
 4. The method of claim 1, wherein identifying the at leastone bottleneck component further comprises at least one of: the computeridentifying at least one component of the one or more components havinga greatest memory usage; and the computer identifying at least onecomponent of the one or more components having a memory usage exceedinga threshold value.
 5. The method of claim 1, wherein identifying the atleast one bottleneck component further comprises at least one of: thecomputer identifying at least one component of the one or morecomponents taking a longest time to complete a task; and the computeridentifying at least one component of the one or more components takinglonger than a threshold time to complete a task.
 6. The method of claim1, wherein identifying the at least one bottleneck component furthercomprises: the computer generating a model for operating the datapipeline having one or more parameters and one or more constraints,wherein the one or more parameters include memory usage and time; thecomputer determining a lowest memory usage and shortest time based onthe model; and the computer identifying at least one component of theone or more components that fails to perform at the lowest memory usageor the shortest time.
 7. The method of claim 1, wherein converting codeof the at least one bottleneck further comprises: the computerdetermining that code of the at least one bottleneck component isincompatible with code of the non-distributed computing environment; andthe computer generating a wrapper function enabling compatibilitybetween code of the at least one bottleneck component and code of thenon-distributed computing environment.
 8. A computer program product formapping one or more components of a data pipeline from a non-distributedcomputing environment to a distributed computing environment, thecomputer program product comprising: one or more computer-readablestorage devices and program instructions stored on at least one of theone or more tangible storage devices, the program instructionscomprising: program instructions to identify at least one bottleneckcomponent of one or more components comprising the data pipeline whenimplemented within the non-distributed computing environment, whereinidentifying the at least one bottleneck component is based on a hybridapproach, wherein the hybrid approach sets thresholds through parametersfor each component to complete a task; program instructions to increasean input to the one or more components comprising the data pipeline;program instructions to determine that a performance of the at least onecomponent is less than proportional to the increase of the input;program instructions to convert data corresponding to the at least onebottleneck component from a format compatible with the non-distributedcomputing environment to a format compatible with the distributedcomputing environment; program instructions to install librariescorresponding to an architecture of the distributed computingenvironment; and program instructions to convert code of the at leastone bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment.
 9. The computer program product ofclaim 8, further comprising: program instructions to measure aperformance of the at least one bottleneck component when implementedwithin the distributed computing environment; and wherein identifyingthe at least one bottleneck component of one or more componentscomprising the data pipeline is further based on the measuredperformance of the at least one bottleneck.
 10. The computer programproduct of claim 9, further comprising: program instructions todetermine whether the at least one bottleneck component is capable ofbeing parallelized; and wherein identifying the at least one bottleneckcomponent of one or more components comprising the data pipeline isfurther based on determining that the at least one bottleneck componentis capable of being parallelized.
 11. The computer program product ofclaim 8, wherein program instructions to identify the at least onebottleneck component further comprise at least one of: programinstructions to identify at least one component of the one or morecomponents having a greatest memory usage; and program instructions toidentify at least one component of the one or more components having amemory usage exceeding a threshold value.
 12. The computer programproduct of claim 8, wherein program instructions to identify the atleast one bottleneck component further comprise at least one of: programinstructions to identify at least one component of the one or morecomponents taking a longest time to complete a task; and programinstructions to identify at least one component of the one or morecomponents taking longer than a threshold time to complete a task. 13.The computer program product of claim 8, wherein program instructions toidentify the at least one bottleneck component further comprise: programinstructions to generate a model for operating the data pipeline havingone or more parameters and one or more constraints, wherein the one ormore parameters include memory usage and time; program instructions todetermine a lowest memory usage and shortest time based on the model;and program instructions to identify at least one component of the oneor more components that fails to perform at the lowest memory usage orthe shortest time.
 14. The computer program product of claim 8, whereinprogram instructions to convert code of the at least one bottleneckfurther comprise: program instructions to determine that code of the atleast one bottleneck component is incompatible with code of thenon-distributed computing environment; and program instructions togenerate a wrapper function enabling compatibility between code of theat least one bottleneck component and code of the non-distributedcomputing environment.
 15. A computer system for mapping one or morecomponents of a data pipeline from a non-distributed computingenvironment to a distributed computing environment, the computer systemcomprising: one or more processors, one or more computer-readablememories, one or more computer-readable tangible storage devices, andprogram instructions stored on at least one of the one or more storagedevices for execution by at least one of the one or more processors viaat least one of the one or more memories, the program instructionscomprising: program instructions to identify at least one bottleneckcomponent of one or more components comprising the data pipeline whenimplemented within the non-distributed computing environment, whereinidentifying the at least one bottleneck component is based on a hybridapproach, wherein the hybrid approach sets thresholds through parametersfor each component to complete a task; program instructions to increasean input to the one or more components comprising the data pipeline;program instructions to determine that a performance of the at least onecomponent is less than proportional to the increase of the input;program instructions to convert data corresponding to the at least onebottleneck component from a format compatible with the non-distributedcomputing environment to a format compatible with the distributedcomputing environment; program instructions to install librariescorresponding to an architecture of the distributed computingenvironment; and program instructions to convert code of the at leastone bottleneck component from a format compatible with thenon-distributed computing environment to a format compatible with thedistributed computing environment.
 16. The computer system of claim 15,further comprising: program instructions to measure a performance of theat least one bottleneck component when implemented within thedistributed computing environment; and wherein identifying the at leastone bottleneck component of one or more components comprising the datapipeline is further based on the measured performance of the at leastone bottleneck.
 17. The computer system of claim 16, further comprising:program instructions to determine whether the at least one bottleneckcomponent is capable of being parallelized; and wherein identifying theat least one bottleneck component of one or more components comprisingthe data pipeline is further based on determining that the at least onebottleneck component is capable of being parallelized.
 18. The computersystem of claim 15, wherein program instructions to identify the atleast one bottleneck component further comprise at least one of: programinstructions to identify at least one component of the one or morecomponents having a greatest memory usage; and program instructions toidentify at least one component of the one or more components having amemory usage exceeding a threshold value.
 19. The computer system ofclaim 15, wherein program instructions to identify the at least onebottleneck component further comprise at least one of: programinstructions to identify at least one component of the one or morecomponents taking a longest time to complete a task; and programinstructions to identify at least one component of the one or morecomponents taking longer than a threshold time to complete a task. 20.The computer system of claim 15, wherein program instructions toidentify the at least one bottleneck component further comprise: programinstructions to generate a model for operating the data pipeline havingone or more parameters and one or more constraints, wherein the one ormore parameters include memory usage and time; program instructions todetermine a lowest memory usage and shortest time based on the model;and program instructions to identify at least one component of the oneor more components that fails to perform at the lowest memory usage orthe shortest time.